Skip to content

add memory keyword and unlimited constant for SET MEMORY LIMIT - #27

Open
puzpuzpuz wants to merge 1 commit into
masterfrom
puzpuzpuz_set_memory_limit_tokens
Open

add memory keyword and unlimited constant for SET MEMORY LIMIT#27
puzpuzpuz wants to merge 1 commit into
masterfrom
puzpuzpuz_set_memory_limit_tokens

Conversation

@puzpuzpuz

@puzpuzpuz puzpuzpuz commented Jun 5, 2026

Copy link
Copy Markdown

Summary

Adds the memory keyword and the unlimited constant so the QuestDB web console highlights and autocompletes the SET MEMORY LIMIT { <size> | UNLIMITED } clause.

  • memorykeywords.ts (structural keyword, alongside the existing limit / set)
  • unlimitedconstants.ts (value literal, alongside none / null)

Both inserted in alphabetical order. yarn lint, yarn build, and yarn test (6681 tests) pass locally.

Context

The SET MEMORY LIMIT clause is introduced by:

  • feat(core): add per-query memory limits questdb#7184 — per-query / per-workload memory limits (OSS)
  • questdb/questdb-enterprise#1039 — per-user / group / service-account memory limits, via ALTER { USER | GROUP | SERVICE ACCOUNT } ... SET MEMORY LIMIT { <size> | UNLIMITED }

Docs: questdb/documentation#462

javier pushed a commit to questdb/documentation that referenced this pull request Sep 14, 2026
## Summary

Documents the new memory-limit feature across configuration, monitoring,
and access control.

**OSS — per-workload memory limits**
- `configuration/cairo-engine.md`: new "Memory limits" section covering
the reloadable workload limits (`cairo.query.memory.limit.bytes`,
`cairo.mat.view.refresh.memory.limit.bytes`,
`cairo.wal.apply.memory.limit.bytes`, plus a pointer to
`cairo.live.view.refresh.memory.limit.bytes`, which the live-views
config page already documents), per-workload breach behavior, and a note
that memory-mapped memory does not count toward a limit. Cross-links
added from the WAL, materialized-views, and capacity-planning pages.
- `query/functions/meta.md`: `query_activity` gains the `memory_used`
and `memory_limit` columns.
- `configuration/materialized-views.md`: documents
`cairo.mat.view.refresh.busy.retry.limit` and
`cairo.mat.view.refresh.busy.retry.timeout`, which govern what happens
after a refresh breaches its memory limit and were previously
undocumented. The breach description in `cairo-engine.md` links to them.

**Enterprise — per-principal memory limits**
- `query/sql/acl/alter-user.md`, `alter-service-account.md`, and a new
`alter-group.md` (+ sidebar): `SET MEMORY LIMIT { <size> | UNLIMITED }`.
- `security/rbac.md`: new "Memory limits" section — how limits resolve,
what a per-principal limit does and does not cover, how to inspect
limits, the `SET MEMORY LIMIT` permission, and `ALTER GROUP` in the
command list.
- `query/sql/show.md`: the `memory_limit` column on `SHOW USERS`, `SHOW
GROUPS` and `SHOW GROUPS <user>`, and `SHOW SERVICE ACCOUNTS` and `SHOW
SERVICE ACCOUNTS <user | group>`.

**Limit resolution — override, not min.** Limits resolve by strict
precedence: a principal's own limit → (users only) the most-restrictive
limit among its groups → the global `cairo.query.memory.limit.bytes`
workload limit. A `0`/`UNLIMITED` value at a level means "not set" and
falls through to the next. A more specific limit, when set, fully
**overrides** the broader one and **binds even when larger**, so a
per-user or per-group override can raise a principal's ceiling above the
workload limit, not only lower it. Service accounts never inherit group
limits. (Earlier drafts of these docs described a most-restrictive "min"
behavior; revised per questdb/questdb-enterprise@fb4e050 "Make
per-entity memory limits override, not min".)

## Corrections made after checking against the implementation

Two statements in earlier revisions of this PR were wrong, and both were
verified against the Enterprise code rather than re-reasoned:

- **`UPDATE` was described as never picking up a per-principal limit.**
False for a non-WAL table. `UpdateOperation.apply()` passes the caller's
own `SqlExecutionContext` to `UpdateOperatorImpl.executeUpdate`, which
calls `queryRegistry.register(...)`, which acquires a QUERY-workload
tracker from that context's security context — so a non-WAL `UPDATE`
**is** capped by `SET MEMORY LIMIT`. An operator sizing one from the old
text would have met an unexpected OOM abort. Only the WAL path (the
default table type) runs under the WAL apply job's tracker. Replaced
with a "What a per-principal limit covers" section that splits the two.
- **`SHOW GROUPS <user>` and `SHOW SERVICE ACCOUNTS <user | group>` were
shown without a `memory_limit` column**, with prose saying so
explicitly. They carry it — `AbstractShowLinkedEntitiesCursorFactory`
adds it to both metadata shapes, pinned by `ShowAclTest`. Since these
are among the results that break a positional client, the omission hid
half the breaking change.

Example tables that disagreed with each other on the same entity's limit
were also made consistent.

## Behaviour the docs previously did not mention at all

- **`COPY ... TO` exports are capped by the issuing principal's limit.**
The export runs under that principal's own context on the query
workload, so a limit must be sized for the largest single thing the
principal runs, exports included — not only for its interactive queries.
- **`SET MEMORY LIMIT` is self-escalating.** It takes no entity name, so
its holder can set *any* principal's limit including its own; combined
with override-not-min, a non-admin holding it can raise its own ceiling
above `cairo.query.memory.limit.bytes`. Documented as a warning,
together with the `GRANT ALL` expansion — a principal granted `ALL`
before the upgrade does not acquire the new permission.
- **The breaking change.** Five results gain a column on upgrade (`SHOW
USERS`, `SHOW GROUPS` and `SHOW GROUPS <user>`, `SHOW SERVICE ACCOUNTS`
and `SHOW SERVICE ACCOUNTS <user | group>`), plus `SELECT *` on
`sys.acl_entities`. Positional clients must be updated; name-based
clients are unaffected.
- **`sys.acl_entities`** stores the value, and is readable only by the
built-in admin — a principal holding `DATABASE ADMIN` is still denied.
- **`memory_limit` means different things per statement**: the
*effective* limit in `SHOW USERS`, the entity's *own* limit in `SHOW
GROUPS` / `SHOW SERVICE ACCOUNTS`, since neither inherits one.
- **An external (SSO/OIDC) user's inherited group limit refreshes at
next login**, not on its current session — the same refresh-on-login
model that already governs group-granted permissions for external users.

## Re-validation against the merged code

Both source PRs merged after the last revision of this doc PR (OSS on
2026-06-20, Enterprise on 2026-09-08), and OSS `master` moved on
afterwards. Every claim was re-checked against merged Enterprise `main`
and OSS `master` (`questdb` submodule pin `fef82615`), and these were
stale:

- **There are four workload limits, not three.** Live views
(questdb/questdb#7461) added `LIVE_VIEW_REFRESH` as a workload with its
own reloadable `cairo.live.view.refresh.memory.limit.bytes`. The
live-views config page on `main` already documents the key, so the
memory-limits section now points to it rather than duplicating it, and
the intro, capacity-planning blurb, and the RBAC "not covered by a
principal limit" list name live view refresh.
- **A mat view refresh breach no longer invalidates the view on first
breach.** questdb/questdb#7275 reclassified a refresh OOM as a
transient, retried failure: the refresh is deferred by
`cairo.mat.view.refresh.busy.retry.timeout` and the view is invalidated
only after `cairo.mat.view.refresh.busy.retry.limit` consecutive
failures (`MatViewRefreshJob.isRetriableRefreshError`, pinned by
`WorkloadMemoryTrackerTest`). A live view refresh breach, by contrast,
invalidates immediately (`LiveViewRefreshJob`). The breach paragraph is
now a per-workload list.
- **`ALTER USER ... SET MEMORY LIMIT` on an external (SSO/OIDC) user is
rejected** (`AccessListModel.setMemoryLimit`: `Cannot set memory limit
for external user`). The docs previously only said such users inherit
from groups; they now say the direct form is refused, on both `rbac.md`
and `alter-user.md`.
- **Coverage note trimmed.** The earlier "coverage is best-effort"
paragraph, with its list of tracked allocation sites, is replaced by the
one fact an operator needs: memory-mapped memory, such as the column
files a query reads, does not count toward a limit.
- **WAL apply limit rarely fires.** WAL apply runs only simple
`UPDATE`s, metadata ALTERs, and commits, so the entry now says the limit
mainly isolates WAL apply from the query budget.
- **Upgrade note.** The `memory_limit` column is added by an automatic
migration (`SysMig6`) on the first primary boot or on promote; an ACL
statement refused with a message naming the `memory_limit` column
migration means the node is not yet migrated. The window normally closes
once WAL apply catches up, so retry first; a persistent refusal calls
for a restart, which re-runs the migration.
- `query_activity`'s two memory columns are `null` for nested
registrations, which share the outer query's budget.

Confirmed unchanged and still accurate: the `SET MEMORY LIMIT` grammar
and permission name, override-not-min precedence and `0`/`UNLIMITED`
semantics (`EntMemoryTrackerProvider.effectiveLimit`), the
`memory_limit` column on all five `SHOW` results including the filtered
forms and the `enabled`/`external_alias`/`grant_option` neighbours
(`AbstractShowEntitiesCursorFactory`,
`AbstractShowLinkedEntitiesCursorFactory`, `ShowAclTest`), the `COPY ...
TO` and non-WAL `UPDATE` coverage, the breach message format
(`Unsafe.checkPerQueryAllocLimit`), reloadability of every limit
(`DynamicPropServerConfiguration`), and the built-in admin being
unlimitable.

## Dependencies

Documents:
- questdb/questdb#7184 — per-query / per-workload memory limits (OSS),
merged
- questdb/questdb-enterprise#1039 — per-principal memory limits
(Enterprise), merged

Web-console syntax highlighting for the new clause:
questdb/sql-parser#27

## Verification

`yarn build`, which this repo configures to **throw** on broken links,
broken markdown links, and broken anchors.

## Review follow-up

A level-2 `/review-pr` pass (six agents, findings verified against OSS
`fef82615` and Enterprise `main`) found no wrong technical claims. Its
major findings are fixed in 4be3ffa: stale `SHOW SERVICE ACCOUNTS`
tables on the grant and revoke assume-service-account pages, the `OUT OF
MEMORY` error tag literal and the `wal_tables()`
`errorTag`/`errorMessage` columns, the `queryId` mapping for
materialized view refreshes, `CREATE GROUP ... WITH EXTERNAL ALIAS`, the
two migration-window refusal messages, the RBAC page description, and
the `query_activity` column order with the missing `is_wal` column.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant